Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning

In this work, we investigate stochastic quasi-Newton methods for minimizing a finite sum of cost functions over decentralized network. We first develop general algorithmic framework, in which each node constructs local, inexact direction that asymptotically approaches the global, exact one at time step. To do so, local gradient approximation is constructed using dynamic average consensus to track variance-reduced gradients entire network, followed by proper Hessian inverse approximation. show under standard convexity and smoothness assumptions on functions, obtained from our framework converge linearly optimal solution if approximations used have uniformly bounded positive eigenvalues. construct with said boundedness property, design two fully methods—namely, damped regularized limited-memory DFP (Davidon-Fletcher-Powell) BFGS (Broyden-Fletcher-Goldfarb-Shanno)—which use fixed moving window past decision variables adaptively approximations. A noteworthy feature these they not require extra sampling or communication. Numerical results proposed are much faster than existing first-order algorithms.